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FOREWORD 



The work described in this _report . w^.^nitiated under program element 63720N, 
project Zli75-PN (Training System Design and ft^nag'ement), subproject 05 (Improvdgfc 
Effectiveness in Course Design, Delivery? and ^Evaluation) and was sponsored by the 
Deputy Chief Qf Naval Operations (Manpower, Personnel, and Training) (OP-01). Addi- 
tional funding was provided by the Chief of Naval Education and Training. 

The objective of the subproject was to develop an empirically-based instructional 
design support system to aid developers in choosing instructional, alternatives based on 
cost/benefits and specific resource limitations. , The objective of this report, which is 
intended for course designers ancl developers and those managing instructional develop- 
ment, was to describe the development and evaluation of the instructional quality 
inyentory (IQI), a systematic methodology for reyiewih^ instructional material. The 
intent is to improve the quality of the materials^ thereby increasing the effectiveness of 
later student tryouts. 

The work on automating the lQI is" being funded under subproject Z1388-PN.01 (Low- 
cost Microcomputer Training Systems). 



J. W. RENARD 
> Commanding Officer 



3 AMES W. TWEEDDALE 
Technical Director 
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Problem 



SUMMARY 



The Navy has adopted the Instructional Systems Design (ISD) technology for the 
development of its training programs. Although there is a general concern ibr the quality 
of the products developed using, this technology, there* are ho techniques for evaluating 
these products prior to conducting student tryouts.' : QecaUse of the expense of mcking 
extensive revisions following student tryouts, the Chief of Naval Education and Training 
(NAVEDTRA I10A, 1981) has directed the designer arid a subject matter expert to perform 
an "internal, review" of preliminary instruction forjudging accuracy* arid completeness, the 
adequacy of the instructional sequence, the "motivating capability* and the support for 
student learning. However, no method is specified for perfoTming this review. 



The objective of this effort was to review the development and evaluation oi the 
instructional quality inventory (IQI), a systematic methodology for reviewing the thr&e, 
major products oi the ISD process— objectives, test items, and instruction— before 
conducting student tryouts. * 

IQI Development a nd Ev al u a tion • 

The IQI was developed to" assist in the formative evaluation of instruction. It was 
tested empirically jn experiments that examined the validity of its individual prescriptions 
and in studies that demonstrated its applicability and utility. In addition, it underwent 
extensive formative evaluation in a series of over 50 workshops. 

Conclusions • 



The IQI is an effective empirically-based methodology for reviewing objectives, test 
items, and instruction for consistency and adequacy. It can. be used by instructional 
development personnel after 'a minimal amount of training. However, it is not designed to 
assist developers in adapting instructional programs to individual student learning styles, 
experiential background, or preferences. Further, it does not address the overall 
structure of the instructional content and how best to sequence the instructional program. 

Recommendations ^ 



It is recommended that instructional developers use IQI to (a) assist in the 
formative evaluation of newly-developed ' instruction, (b) review existing instruction for 
consistency and adequacy, and (c) provide standards for contractor-developed instruction. 

2. It is recommended that IQI procedures be automated to (a) minimize skill 
deficiencies in instructional development personnel, (b) allow designers to pay more 
attention to the difficult tasks of analysis and design, and (c) speed ■ the internal review 
process and facilitate revisions. 
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INTRODUCTION 



Problem 

The Navy _ has adopted the Instructional Systems Design _(ISD) technology for the 
development of its training programs (Branson, Rayner, Cox, Fur man, King, <5c Harnum, 
1975; Branson, 1977, 1979), Although there is a general concern for the quality of the 
products developed using this technology, there are* no techniques for evaluating these 
products, before conducting student tryouts. Because of the expense of making extensive 
revisions following student tryouts, the Chief of Naval Education and Training (NAVED- 
TRA 1 10A, 1981) has directed the designer and*€t subject matter expert (SME) to perform 
an "internal review" of preliminary instruction for judging accuracy and completeness, the 
adequacy of the instructional sequencfe, the motivating capability, and the support for 
student learning. However, no method is specified for performing the review. 

Objectiv e 

_ - a __ _ z 

The objective of this report is to describe the development and evaluation of the 
.instructional quality inventory (IQI), a systematic methodology for reviewing the three 
major products of the ISO process — bbjectives, test items, and instruction — before 
conducting student tryouts* 



IQI DEVELOPMENT AND EVALUATION 

Background 

Current approaches tp the management of instructional design and development 
combine methods" for systems analysis or operations research with learning and instruc- 
tional psychology. The armed services have all adopted the ISD methodology for 
accomplishing instructional design and development (Branson et ah, 1975; Branson, 1977, 
1979). Although there are weaknesses in the proceduralized guidebooks that support ISD 
and its management (Montague & Wulfeck, 1982; Vineberg 3c Joyner, 1980; Wetzel, Ellis, 
Wulfeck, & Montague, 1982), they do prctvide basic guidance for analyzing, designing,, 
developing, evaluating, and implementing instruction. Since all instruction, whether 
developed formally or informally, goes through those stages or phases, it is generally 
agreed that this focus on task-referenced instruction and the exploitation of modern 
communications technology for instruction is a desirable goal (e.g., Dick & Carey, 1978; 
Gropper, 1980; Gagneoc Bi iggs, 1979, O'Nell, 1979a, 1979b). 

A major problem with ISD is ^^_ii_^P eri ^ s _ PA the designer/developer's expertise. 
How ^vell the stages are parried out jtepends, in large part, on the organization, artistry, 
and effort put into them. ^ ^rriejlmes variou^s steps are left oat or ignored, "new 11 
instruction adopts old materials by default, tests ar e inconsistent with instructional 
requirements, materials are inadequate, etc. in general, ISD is long on "what to do" but 
short on "how to do it." For example, at several points in the design sequence, 
intermediate products are developed, such as objectives, test items, and segments of 
instruction to support learning. Although there is a general concern about the quality of . 
the products being developed, no techniques are given for evaluating them before student 
tryouts are conducted. * 

In fact, major guides for design and development _ describe formative evaluation 
almost exclusively in terms of such tryouts (e.g., Dick, 1977; Gagne& Briggs, 1979; Bloom, 



Hastings, & Madaus, 1971; Tuckman, 1979). Since the design process is based on. 
approximate and imperfect knowledge, such tryduts are vital to detect inadequacies in the 
materials, for example, in his widely cited chapter, Dick (1977) describes three stages or 
phases of formative evaluation. First, lesson materials are tried with students one-at-a- 
time, so that glaring faults and misunderstandings can be correctedr Second, small group 
trials are held, and further revisions are made as necessary. Third, the prototype is tested 
in the standard environment under the appropriate operating conditions. Final revisions 
are "based on various forms of information^ including test -performance, student and 
instructor questions and comments, etc. Because of the expense of this process, internal 
review has recently received attention. 

Merrill and his associates (e.g., Merr'U <5c Bout well, 1973; 'Merrill, Richards,' Schmidt, 
<5c Wood, 1977),- who were working on the problem of standards for the instructional 
development process, were initially contemned with deriving instructional prescriptions 
from research studies on instructional variables: They suggested that such prescriptions 
served ncrt only to recommend instructional practices but also^is'a basis for reviewing 
existing instructions for internal consistency and adequacy. Subsequently, they developed 
the instructional strategy diagnostic profile (ISBP) (Merrill et al.,_ 1977). The ISDP 
procedures underwent extensive field testing and revision at the Navy Personnel Research 
and Development Center, and resulted in the IQI. Merrill, Reigeluth, and Faust (1979) 
described a slightly different version of the procedures, and Reigeluth (1980) suggested 
using them to train teachers to improve classroom teaching. 

IQI Classification Scheme 

IQI was designed initially to parallel and supplement the military's ISD mocJel. 
However, it can be applied to any systematically developed program of instruction that 
has objectives, test items, and instruction tied to the objectives, since it focuses on those 
three rtmin parts. It is intended for use with cognitive and psychomotor instruction and is 
appropriate for evaluating most forms of such materials (cf ., Roid <5c Haladyna, 1982). 

IQI use's a scheme that classifies objectives, test items, and instructional presenta- 
tions. Classification is useful for several reasons: 

, \ 

1. It helps make more precise judgments about the adequacy of learning objectives 
arid leads to more precise test item specifications. 

2. It assists the internal review process by gyiding judgments about the consistency 
between objectives and test items. If objectives and test items were not classified, all 
one could say is "This is an objective and this is a test item; and they don't look too 
different. 11 

3. It not only ensures that objectives, test items, and the corresponding instruction 
address the same learning arid performance requirements, but also helps to judge whether 
or not they are adequate. 

Classification schemes have been used in the past with some success. However, if a 
classification scheme is to be useful, it must meet two conditions. First, instructional and 
test developers must be §ble to make reliable classifications using the scheme. Second, 
the scheme must have clear implications for instructional and test item development; the 
implications should be specified as prescriptions for development. Typically, classifica- 
tion procedures are too loosely defined (e.g., Gagne , 1976; Popham, 1978) to permit 
reliable classification by users who have-not had considerable training or who are not 



highly sophisticated instructional designers; " Although military instructor personnel have 
clifficiilty in using schemes of this nature, they do the majority of instructional and test 
development. In addition, the* implications of classification for instruction arid testing are 
often hot clearly specified (Briggs,_ 1977). The I§r classification scheme has been tested - 
and extensively revised (Ellis & Wul£eck, 1979) arid, is currently . in wide use by Navy 
personnel. In its present form^ it meets the two conditions listed afc>ove (Ellis <5c Wulfeck, 
1982)- :: ' - 

N 

; r . . ..... \ 

Objectives and test items can be classified according to: (1) what the student must 
do (i.e., the task to be, performed), and (2) the instructional content (i.e., the type of 
47 information the student must learn), the IQI task/content matrix is displayed in Tabl£ 1 
and discussed below: ; 1 . 

1. The T a sk D i mension . A student can either remember information, or use /the 
information- to .do something. This distinction coyesponds to the difference between 
knowledge and application and to the difference between declarative and procedural 
knowledge (Ryle, 1949; Broudy, 1977). The use level can be further divided into two types: 
(a) use-unaided , where the person has no aids except his or her own memory, and (b) use- 
aided , where the person- has a job aid designed to reduce dependence on memory.. 

2. The Content Dimension . There are five content types: 

a. Facts, which are simple associations between names, objects, symbols, 
locations, etc., can only be remembered while the other content types can be remembered 
or -used, v - - J_ 

b. Categories are classifications defined by certain specified characteristics. 

c. Procedures consist of ordered sequences of steps or operations performed on 
a single object or in a specific situation. 

d. Rules also consist of ordered sequences of operations but can be performed 
on a variety of objects or in a variety of situations. 

* *■■_____ . ... <> 

s e. Principles involve explanations, predictions, or diagnoses based on theoret- 
ical or cause-effect relationships.^, 



Tabic 1 


Task levels 


Content Types 


» — i 


Fact 


- Category 


Procedure 


Rule 


Principle 


Remember 


Recall or 
recognize, 
names, partsr 
jtfateSTetc. 


State the 
basis for clk> 
sifying the 
characteris- 
tics. 


List steps 
used. 


State the 
rule that 
applies to 
these situa- 
tions. 


State the v « 
principle that 
explains 
these events. . 


j Use-unaided 


Not applica- 
ble. 


Classify 
objects, 
events, by 
characteris- 
tics. 


Do steps in 
this context 
or on this 
equipment. 


t^se rule or 
formula on 
these types 
of tasks or 
ne.w prob- 
lems. x - 


Explain ox 
predict 
event (s) 
using princi- 
ple. 


Use-aided 


Not applica- 
ble. 


Given 
characteris- 
tics, classify. 


Given steps, 
do them. 


Given for- 
mula, use to 
solve. _ . 


Given princi- 
ple, use to 
explain^ 



In summary, the remember level involves simply remembering facts arid the steps> 
descriptions^ or definitions of categories, procedures, rules, and. principles. • The User 
•unaided level involves remembering what is to be done and then doing it. The use-aided 
level involves doing tasks usin§ a jph aid. _ 

1QI Procedures 

The IQI consists of five procedures, which are listed in Table 2. : 



Tabic 2 

IQI EVALUATION PROCEDURES 


instructional component 


Procedures 


Classified. Objectives 


.". , 

1. Judge adequacy of objectives/goals. "> 

< 


Tests and test items 


2- Are objectives and test items consistent? 
3. Are test items adequate? * • 


Instruction/Presentations 


4. Are prestations consistent with objectives? 

5. Are presentations adequate? 



Since all the steps of the instructional development prpcess depend on careful 
specification of learning objectives, the first procedure is to assure the adequacy of 
objectives. This is done by classifying each objective, deter m^ing whether all the 
necessary conditions, standards, and actions are present, and. judging whether or not the 
objective accurately reflects what the student is supposed to do or know following 
training. * " 

The next two procedures involve checking test items. First, test item(s) are 
evaluated against their corresponding objectives to ensure that the Jest items are 
consistent with the objectives. This is accomplished by making sure that the conditions, 
standards, actions, task level, and content type for .the test items are the same as those 
for the objective.. Consistency problems occur frequently, even in systematically 
developed instruction. It is likely that restrictions (e.g., in testing time, laboratory space, 
etc.) can cause instructors to develop inconsistent tests. For example* Lockhart, Sturges, 
Van Matre, and Zachai (1981) found that 79 percent of the test items in a recently 
developed Navy technical course were faulty, and over 40 percent of these involved 
inconsistencies between objectives arid test items. Next, the adequacy of the test items 
Is assessed by determining whether each item conforms to criteria 'for proper item 
construction. ' 

At this point, the objectives and test items for the instructional program are 
consistent with each other' and are adequate. The next step is to ensure that the 
instructional presentation (e.g., printed self-study materials, lectures, computer-assisted 
instruction, films, etc.) is consistent with the objectives and test it<?ms; that is, it must 



teach to. the task level, arid .content type of the objective/test item. Different 
combinations of instructional components (e.g., expository statements* example:^, 
* practice * feedback)* "are required, depending _ on the task level of "th€t_ objective. A 
component is counted as present in the. instruction only if it is complete. This depends on 
the content type of the objective and simply means that everything that needs tb be 
/included is included* % " 

Since instruction can be consistent but still not teach "as effectively as it could, the 
final procedure evaluates the adequacy of the instructional presentation. The adequacy 
judgment incorporates a number of instructional design principles that have been shown to 
promote student learning, including guidelines for formatting information so students can 
* find it, communicating it clearly and effectively so students, ban understand it, and * 
providing additional explanation, so students can better learn and retain the. information. 

The IQI consists of tj^e following four volumes, which: are available from the Defense 
Technical Information Center, Alexandria, Virginia (see references): 

1. . ' Introduction and bverview^Wuifecl^, Ellis, Richards, Wood, <5c Merrill, 1978). 

2. Users' Manual (Ellis, Wulfeck, <5c Fredericks, 1979). «• \ . ' 

3. Training Workbook (Fredericks^ 1988). .* • ' • 
k. Job Performance Aid (Ellis <Jc Wulfeck, 1978). 

Empirical Support for IQI 

— _ -s - ' _ ___ " ____ _ 1 " - _ -» S_ 

. • This section review's some of the experimental tests of the IQI prescriptions, some % 

attempts tb demonstrate its Utility and validity in diagnosing and revising inconsistent and 
inadequate instruction, and its own formative evaluation. 

Empirical Studies / 

/' - ----- ^ , . : 

; Merrill, Reigeluth, and Faust (1979), in reviewing existing pertinent research, found- 

considerable support for IQI .prescriptions. Of 51 studies that were -.valid tests of IQI 
principles, none contradicted any of its principles. In addition, they conducted a number 
of empirical studies to test (1) Jthe consistency between instruction and test items, (2) 
prescriptions* about the separation of statements, examples, and practice, and/or (3) the 
effects of attention- focusing helps for classification tasks, elaborative helps for state- 
ments and examples, and instance divergence and matching for classification tasks. 
. Results of all studies supported IQI prescriptions and principles. - - 

jMerrili^ Wood, Baker, Ellis, Jm^ study to ^ errfpi dealt y in- 

validate some of ^the consistency and adequacy prescriptions of th^ IQI. Specifically, the 
study was designed to test six hypotheses concerning the consistency and adequacy -of 
instructional materials. Subjects participating in the sttidy were^U.S. Navy enlisted men 
waitipg, 'to begin "A" School at the Propulsion Engineering (PE) -School,, Grept* bakes; 
Instructional materials developed were based on PE school curriculum but were adapted to 
provide*for 12 experimental treatments needed to test the six hypotheses. Four of these 
treatments represented remember-level instruction; and eight, use-level instruction. 
After students finished -their instruction, they were tested; on remember-level test items 
(labeling and listing) and use-level test items (classification). All subjects had the same 
testing materials. Three experiments were conducted, which are described below. 

\.~ Experiment J 'tested the consistency hypothesis (i.e., performance will- decrease * 
if test items and^ the instruction are not consistent) by manipulating test items and the 
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"instruction. Results showed that the students in the use- level treatment groups scored 
significantly higfter on use-level items (classification) than did those in the remember- 
level groups, and students in the remember-level groups scored significantly higher on 
remember-level items (labeling and listing) than did those in the use-level groups. Thus, 
the consistency prescription of the IQI was supported. 

2. Experiment II tested the adequacy hypothesis for -remember-level items (i.e. , 
performance will increase with the use of a mnemonic or several-page distributed 
practice) by. comparing the performance of jaj a mnemonic ^ group and J*_ no-mnemonic 
group and (b) a several-page jdistributed-practice group ^P^.^Pr^P^^.^^^-practice 
group. Results showed that there were QP.^ign^fi? 3 ^ J^i*? 1 "^! 10 ? 3 . in Performance on the 
remember-level ^test items \or either the jrinemoniq or practice variables, but there was a 
significant savings of time for the massed-practice condition. Thus, the two presentation 
adequacy prescriptions of the IQI for remember-level items were riot supported. 

1^3. Experiment III tested the adequacy hypotheses for use-level items (i.e., perfor- 
mance will increase with the use of definitions that are separated from the rest of the 
instructional presentatioh^ivergent examples, and attribute, isolation elaboration). This 
was done by comparing 'the performance of (a) a separated-defnition group and an 
embedded-definition group, (b) a divergent-example group and a convergent-example 
group, and (c) an attribute-isolation group and a no-attribute-isolation group. Results 
showed that students given separated definitions scored higher on all performance 
measures and took less time than did those in the embedded-definition treatment. 
Students in the divergent-example treatments scored higher on use test items than did 
those in thfe convergent-example treatment. There were no differences between students 
in the attribute-isolation and no-attribute-isolation elaboration treatments. Thus, only 
two of the three presentation adequacy prescriptions of the IQI for use-level items were 
supported. \ % 

. Demonstration Studies • . / ■ 

ess the IQI's usefulness for revising existing 
the effectiveness of the original and revised 

Study I. IQI procedures were applied to instructional materials taken from a Navy 
radioman course that described several types of Navy call signs (call signs are used to 
identify radio stations (e.g., WABC or KNBG)), and a revised version was developed. The 
IQI and school versions were compared using two groups of 20 Navy subjects. The 
objectives for the lesson required students to (1) recall the names and defining character- 
istics of five types of Navy call signs (remember-category. task), and (2) to classify new 
call signs according to one of the five types (use-category "task). After completing the 
materials^ study time was recorded, and subjects were asked to recall a written list of the 
names "and definitions of the call signs and to classify a list of IS call signs. Dependent 
measures included, the times required, to complete the materials and tests and the number 
corect on the tests. A 

■H _ _ _ _ . 

'.Results showed that IQI subjects performed significantly better-on both tests than did 
school subjects. Means for the IQI group, for the recall and classification tests were 83 
and 81 percent respectively, compared to 49 and 60 percent for the school group. There 
were no differences in time to compjjete the materials but IQI subjects took less time to 
complete the tests. 



Two studies were conducted to 
segments of instruction and to com par 
materials. 



Study II . The objectives of this research were to (i) determine whether relativeiy 
inexperienced persons could use IQI to identify difficulties in and revise instructional 
materials, (2) evaluate the effectiveness of the revised materials, and (3) determine costs 
of applying the IQI (Stern Sc Fredericks, 1982). A lesson for Navy radiomen, which trains 
students to verify spelling, punctuation, arid character aligrimerit on a special optically 
scanned form, was selected since it had been identified by instructors as needing revision. 
A two-person team with limited expedience in the use of the IQI process analyzed the 
selected material, and examined the objectives, testing format arid content, and training 
materials. This resulted in (1) some changes to the objectives, which were found to be 
inconsistent with job requirements, (2) an entirely different approach to testing, and (3) a 
new set of training materials. During the process, lessonware development costs were 
tracked to allow for later cost analysis. 

To evaluate the effectiveness of the revised materials, two groups of 30 students 
were compared on learning from the old. and revised lesson. Results showed that the 
groups differed significantly on thejr tgst performance. - The average score for IQI 
students was 57 percent correct, compared to 49 percent for standard students; The 
standard group had studied their materials for an average of 11.6 hours, compared to 2*3 
hours for the IQI group. Thus, the IQI group scored somewhat higher on the job-relevant 
test than did trainees in the standard group and learned more efficiently. In addition to 
the empirical findings, the IQI provided a useful framework for determining deficiencies 
in current instruction. By examining module components in the sequence called for by 
IQI, deficiencies could be readily identified and revisions made. Finally, .an important 
consideration in revising instruction concerns the costs incurred in the process. These 
were almost all in hourly costs for the research team, amounting to one-fifth of a man- 
year. However, it is estimated that this figure would be considerably reduced if IQI were 
applied concurrently with the ISD process and would be less as the team gained 
"experience. The findings suggest that the IQI material is more effective in producing job- 
relevant performance than are existing formative evaluation methods. 

Formative Evaluation Workshops 

In addition to the empirical studies, the IQI procedures were subjected to a rigorous 
formative evaluation. A series of over 50 workshops involving several hundred potential 
users were conducted .(Wood, Ellis, & Wulfeck, 1978). Participants in these- workshops 
were asked to comment on and criticize the validity and utility of the IQI procedures. 
The information gained from conducting the workshops was used to revise and improve the 
IQI. 




DISCUSSION AND CONCLUSIONS 

Usefulness arid Validity 

The main contribution of the IQI is that it synthesizes research knowledge on 
instr uction in a form that is implementable. The underlying principles are not new. They 
were derived from the general research literature and we re confirmed by studies^ 
-specifically designed -td^^t "found that 

contradict the bases for quality assessment in the IQI, it appears that its underlying « 
conceptions are sound. 



The demonstration studies attempted to use the IQI in operational environments. 
Study I demonstrated that experienced personnel could revise existing course materials 



with corresponding significantly increased student test performance ^jUgg** 
Fredericks 1982) demonstrated the effectiveness of the efforts ot less experiencea 
nersonnel The IQI is an effective tool that can be used with a limited amount of training 
¥S ma^VelsoVfbr its acceptance appears to be that it provides a systematic framewo^ 
for applying current knowledge about instructional strategies and for managing mstruc- 
tional program development. 

The IQI framework has also proven useful as a basis for guiding , the construction of 
tests of student learning. Performance-based training requires the use of tests that are 
criter°on!referenced; that is, that directly measure aspects of performance required in 
%£ em£:tw™l* addition, during instruction, it is important to ^ able to de er n 
whv a student cannot perform well so that the problem or errors can be corrected. 
ma R nos? c tests give information about gaps in student knowledge or skill that serve to 
Side SmediaSn. Although both criterion-referenced and diagnostic tests are needed in 
Pn^tructiOrT Smatic guidelines for their development were incomplete or nonexistent 

n ISD procedurls (Wetzel, Ellis, Wulf eck, «5c Montague, 1982). The IQ classifica- 

tion scneme (see Table 1) provides a means of classifying objectives that has clear 

mplicatfons for test development. These implications have been specified as pr«^rip- 
Sfor test development, and guidelines have been prepared f or their otd gfg^g^ 
Wulf eck 1982; Roid & Haladyna, 1982). A series of user workshops provided £jm?gY? . 
e^ahStlon Tot 'the guidelines, and they are being adopted for use by the Naval Education 
and Training Command. 

Limitations^and Need for Additional Develop ment 

The iOi is limited in what it attempts to accomplish. There are some important 
characteristics of instruction to which the IQI does not attend. One concerns adapting 
'nSucfiona praams to individual student learning styles, experiential background or 
preferences^. Glaser (1977) discusses . the theoretical issues but provides little direct 
guidance for design. 

Another problem centers around the analysis of content structure. Systematic 
anoroaches to instructional design emphasize the identification of requirements- in the 
fSS ^M^*iS%ecti« g ftiS6ugh this was an important ^nce in a e t 
to develop efficient and effective training, it also resu ts in decontextualization. An 
exhaustive detailed, linear progression through the objectives is the form taken by much 
of the fnTtrucdon (Smith & Reigeluth, 1982). Important relationships between topics are 
ignored, and ro? e memorization^ unorganized^facts is promoted A student's processing 
capacity is taxed and little or no context is provided for retrieval of needed information 
or skill. The IQI was not designed to handle such problems. 

Reigeluth, Merrill, Wilson, and Spiller (1980), in developing the elaboration theory of 
instruction attempted to provide prescriptive recommendations for use in organizing and 
sSuctur n"' performance objectives. Better structure and sequence should improve 
learning efficiency and retention, it is important to begin incorporating these ideas into 
existing evaluation procedures. 

r 

Need for Automation 

The focus on the consistency and adequacy considerations are important and- can be 
used to correct very prevalent errors in instruction. Even with the revisions of the 
m*h£olw^wer? Wwrients about adequacy rely considerably on the knowledge of 
^S^'iSScs'tad Of relevant psychological and educational research. Since 



training programs for this knowledge are very rare, the skills are likely to be rare among 
those persons who are__responsibIe for instructional development. Although techniques 
such as those in the IQI could be expanded considerably with increasingly more detailed 
procedures to overcome these skill deficiencies, they would be less likely to be used. The 
key appears to be automation. Computer-based aids can reduce the apparent complexity 
arid take on more mundane tasks, allowing designers to pay more attention to the difficult 
tasks of analysis and design. An automated version of the IQI could speed the internal 
review process and facilitate corrections. Automated aids for instructional development 
and evaluation are currently being developed, 

RECOMMENDATIONS 

_:_ 1. It is recommended that instructional developers use IQI to _ in_ the 

formative evaluation of newly-develpped instruction, (b) review existing instruction for 
consistency and adequacy, and (c) provide standards for contractor-developed instruction, 

2. It is recommended that IQI procedures be automated [ to (a) m^tnimize _skiii 
deficiencies in instructional development personnel, (b) allow designers to pay more 
attention to the difficult tasks of analysis and design, and (c) speed the internal review 
process and facilitate revisions. 
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